A common problem facing information security personnel is the need to identify suspicious or outright malicious software or data on a computer system. This problem typically arises when a computer system is compromised by an attacker through a piece of malicious software. Initial steps taken in response to this kind of situation include attempts to identify malicious software (also known as “malware”) or data, followed by attempts to classify that malicious software so that its capabilities can better be understood. Investigators and response personnel use a variety of techniques to identify suspicious software, such as temporal analysis, filtering of known entities, and Live Response (described below).
Temporal analysis involves a review of all activity on a system according to date and time so that events occurring on or around a time window of suspected compromise can be more closely examined. Such items might include event log entries; files created, deleted, accessed, or modified; processes that were started or terminated; network ports opened or closed, and so on.
Additionally a comparison of files against known entities can be performed. In this situation, all files on the system may be reviewed and compared against a database of known, previously encountered files. Such comparisons are usually accomplished through use of a cryptographic hash algorithm—a mathematical function that takes the data from a file and turns it into a compact numerical representation. A fundamental property of hash functions is that if two hashes generated using the same algorithm are different, then the data used to generate those hashes must also be different. The corollary is that hashes found to match were generated from data that was identical. While the corollary is not always true, hash collisions (identical hashes generated from different input data) for cryptographic hash algorithms are provably rare such that a hash comparison can be used to determine file equivalence.
An alternative to reviewing static historical data such as files and event logs is Live Response. This technique examines running programs, system memory contents, network port activity, and other system metadata while the computer system is still on and in a compromised state in order to identify how it may have been modified by an attacker.
There are many other techniques that may be employed to identify suspicious activity on a potentially compromised computer system. These techniques often generate a rather large amount of data, all of which must be reviewed and interpreted in order to reach any conclusions. Further complicating this equation is the fact that attackers typically have a good understanding of the techniques used to identify compromised systems. They employ various methods to hide their presence, making the job of an investigator that much more difficult. Some of these techniques include deleting indicators of their entry to a system once it's compromised, such as log file entries, file modification/access dates, and system processes. Attackers may also obfuscate running malware by changing its name or execution profile such that it appears to be something benign. In order to better hide malware or other data stored on disk, attackers may make use of a “packed” storage format. Packing is a technique by which data is obfuscated or encrypted and encapsulated along with a program to perform a decryption/de-obfuscation, and then stored somewhere on a system. For example, a “Packed Executable” is a piece of software that contains an “unpacking” program and a payload. That payload is often malicious software, such as a virus or Trojan Horse.
One of the fundamental properties of encrypted, compressed, or obfuscated data (depending on the method of obfuscation) is its entropy, or randomness, tends to be higher than that of “structured” data, such as user generated documents and computer programs. A measure of entropy isn't a guaranteed method for identifying malware or an attacker's hidden data store. A valid system user may have encrypted, or more commonly, compressed, information stored on a computer system. However, the examination of entropy does provide an excellent filter for this significant data reduction problem. Entropy is a measurement that can be used to determine if a stream of data is random, provided it is comprised of a defined set of data values. There are drawbacks to using entropy across a block of data, though. Entropy is a global measurement across a data set, returning a single value across that set. This means that a data block could return a low entropy measurement when in fact small sections of that same data could contain very high entropy. This scenario could be true even if the majority of the data block has low entropy. This may be noteworthy, depending on the expectation of the contents of the data. For example, if an attacker has placed an encrypted copy of malware inside of a more structured set of data, the variance of entropy across that otherwise structured data may be a clear indicator of malware. Thus, there is a need in the art for a technique to derive a robust measurement of entropy in order to detect the presence of malware in a computer system while reducing the number of false positives generated during the detection process.